Measurement issues in the evaluation of projects in a project portfolio

A common problem in multicriteria decision analysis is how to value projects in a project portfolio. Recently, there has been some attention given to a practice whereby analysts define a value function on some criterion by setting 0 as the value of the worst project. In particular, Clemen and Smith have argued this practice is not sound as it gives different results from the case where projects are "priced out", and makes a strong implicit assumption about the value of not doing a project. Moreover, it can be shown easily that the optimal choice set is not invariant with respect to rescalings such as those induced by the addition of a new item to the choice set, hence rank reversal can occur when this procedure is used. We provide a measurement theoretic account of the phenomenon and show that whether the admissible transformations of the underlying scale are similarity or affine (or other) transformations depends on the precise details of the value model used. We also provide an axiomatic impossibility result which illustrates an incompatibility between an idea that inaction has a value of zero and the admissibility of constant scale translations. We use our analysis to comment on the view of Clemen and Smith that the baseline problem is best dealt with by assigning project-specific scores for not doing particular projects

Maximising the Benefits of Foreign Aid : Leveraging In-Country Financing

This Policy Brief outlines an alternative approach to maximising the benefit of donor aid in low income countries. It has policy implications for the allocation of aid by Non-Governmental Organisations (NGOs) and national governments

Complementary logics of target-setting : hierarchist and experimentalist governance in the Scottish National Health Service

Where policy ends are contested and means for change are ambiguous, imposing central targets on local organisations – what we call hierarchist governance – is problematic. The concept of experimentalist governance suggests that target-setting should rather be a learning process between central regulators and local organisations. However, the relationship between experimentalist and hierarchist governance remains unclear. Existing literature suggests that the learning-oriented experimentalist logic is hard to reconcile with a hierarchist logic focussed on accountability for results. We examine whether complementary use of hierarchist and experimentalist ideas is possible. Drawing on experiences from Scotland, we find that experimentalist and hierarchist logics can co-exist in the same performance management system. Each logic served distinct roles with respect to target-setting, implementation and accountability. The emphasis on experimentalism was stronger where ends and means were contested (the case of shifting the balance of care for older people) than where both ends and means seemed obvious initially (the case of healthcare-associated infections, where target-setting followed a more hierarchist logic). However, governance drifted towards experimentalism when rising rates of community-acquired infections decreased clarity about effective interventions. The nature of policy issues and changes therein over time appear to be important conditions for synergies between governance logics

Efficiency measurement for management

The previous chapter has discussed the use of efficiency analysis tools to guide policy development and formulation. While few readers will doubt that clear, consistent policy direction is necessary for the delivery of productivity improvements, it is not sufficient. To lead to action on the ground, policy interventions have to influence the behaviour of the staff who see and treat patients, and deliver public health and social care programmes. In this chapter, we discuss the challenges facing management as it seeks to use the analytic tools discussed elsewhere in this volume to secure efficiency improvements

Strategic appraisal of environmental risks: a contrast between the United Kingdom's Stern Review on the Economics of Climate Change and its Committee on Radioactive Waste Management

In this article, we compare two high-profile strategic policy reviews undertaken for the U.K. government on environmental risks: radioactive waste management and climate change. These reviews took very different forms, both in terms of analytic approach and deliberation strategy. The Stern Review on the Economics of Climate Change was largely an exercise in expert modeling, building, within a cost-benefit framework, an argument for immediate reductions in carbon emissions. The Committee on Radioactive Waste Management, on the other hand, followed a much more explicitly deliberative and participative process, using multicriteria decision analysis to bring together scientific evidence and stakeholder and public values. In this article, we ask why the two reviews were different, and whether the differences are justified. We conclude that the differences were mainly due to political context, rather than the underpinning science, and as a consequence that, while in our view “fit for purpose,” they would both have been stronger had they been less different. Stern's grappling with ethical issues could have been strengthened by a greater degree of public and stakeholder engagement, and the Committee on Radioactive Waste Management's handling of issues of uncertainty could have been strengthened by the explicitly probabilistic framework of Stern

The Scottish NHS : meeting the financial challenge ahead

The Scottish NHS faces a crisis of affordability in the next couple of decades as the population ages and demands on services intensify. This presents two challenges: the first is how to redesign services to achieve greater efficiencies, and the second is how to engage the public so that there is a realistic public view about what is affordable, against which a mature discussion about the hard choices about funding and provision can take place. We refer to these as the innovation and openness challenges. In the paper we outline the current state of the system and discuss possible policy options. We conclude with some recommendations for next steps

Ageing, health status, and economic activity in Scotland : a twenty year view

Countries worldwide face the economic and public expenditure challenges of an ageing population. However, ageing per se is but one part of the challenge. The forecast for Scotland’s population health and labour market status will further impact on the scale of required public spending – in health and social protection - as well as on the base of those who are economically active and capable of bearing the scale of such expenditure (by way of taxation). This analysis points to the value on the public expenditure side of early health, education and social protection interventions to reduce future forecast expenditure in health and social protection. On the revenue raising side, it points to the need to expand Scotland’s future effective working age population by way of reducing those considered NEET, expanding female and post age 65/67 labour market participation, and of attracting new working age migrants to Scotland

Multiattribute value elicitation

Multiattribute Value Theory (MAVT) methods are perhaps the most intuitive mul-ticriteria methods, and have the most theoretically well-understood basis. They are employ a divide-and-conquer modelling strategy in which the value of an op-tion is conceptualised as a function (typically the sum) of the scores associated with the performance of the option on different attributes. This chapter outlines the concept of preferential independence, which has a critical underpinning role of elicitation within the MAVT paradigm. I also present MAVT elicitation in the context of the overall Decision Analysis process, comprising three broad stages: Designing and Planning; Structuring the Model; and Analysing the Model. I out-line some of the main practical methods for arriving at the partial values and weighting them to arrive at an overall value score, including both traditional methods relying on cardinal assessment, and the MACBETH approach which uses qualitative difference judgements. A running example of a house choice problem is used to illustrate the different elicitation approaches

Treacle and smallpox : two tests for multi-criteria decision analysis models in health technology assessment

Multicriteria Decision Analysis (MCDA) is, rightly, receiving increasing attention in Health Technology Assessment. However, a distinguishing feature of the health domain is that technologies must actually improve health, and good performance on other criteria cannot compensate for failure to do so. We argue for two reasonable tests for MCDA models: the treacle test (can a winning intervention be incompletely ineffective?) and the smallpox test (can a winning intervention be for a disease which no one suffers from?). We explore why models might fail such tests (as the models of some existing published studies would do) and offer some suggestions as to how practice should be improved